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(54) Method for positional matching of images and system for testing products by using the same 

(57) The present invention provides an image 
matching method which allows non-linear image posi- 
tional shifts to be corrected on a real time basis, and an ^ 
economical product testing system using this method. ^, 
A two dimensional test image, consisting of pixels ar- 
ranged as a matrix of bits distributed in both the vertical 
direction and the horizontal scanning direction, and an 
associated reference image are subjected to template 
matching at a cell matching arithmetic unit for each of 
line blocks consisting of a plurality of scan lines. The 
result of the template matching are quantitatively eval- 
uated at an evaluation unit as being good or bad, and 
positional shifts are sequentially computed according to 
good evaluation results. Based on the positional shifts 
thus obtained, the comparative pixel addresses are ac- 
cordingly corrected before the two images are com- 
pared at the comparator. 
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Description 

TECHNICAL FIELD 

5 The present invention generally relates to a system for testing (inspecting) the external appearance of products 

by using image processing technology, and in particular to a method for compensating for positional shifting of one of 
two images with respect to the other by capturing the two images as groups of pixel data given as gradation values 
(so-called gray scale image values) or binary data, and computing positional shifting of image patterns between the 
two images according to such data so that it may be applied to systems for measuring positional shifting between 

fo images, and to a system for testing products based on this method. 

BACKGROUND OF THE INVENTION 

The recent development in image processing technology and semiconductor technology has accelerated the au- 
f5 tomatization of the visual inspection tests of printed matters, and a number of testing systems based on the comparison 
of gradation image data have already been put into practical uses. The basic principle of the comparison method 
comprises the steps of registering an external picture image of a good product as a reference for comparison, obtaining 
a difference of each pixel of the picture image of the product to be tested from that of the reference image, and deter- 
mining the acceptability of the product according to the magnitude of the differences. 
20 It has also been proposed to compare consecutive the picture images of the products which are continually captured 

in time sequence. 

According to these conventional methods, because the working principle is based on the evaluation of the differ- 
ences between the two images in either case, it is essential to be free from any undue positional shifting between the 
picture patterns of the two images. If there any significant positional shifting, it results in amplified difference values, 

25 and the product is inevitably judged as defective. Thus, so-called pseudo defects would occur so frequently that a 
satisfactory testing procedure could not be carried out. 

Therefore, previously, frequent occurrences of pseudo defects have been avoided by computing positional shifting 
between the two images with image processing means, and positional corrections have been conducted on the images 
according to the detected positional shifting between them before comparing the two images. A number of methods 

30 have been proposed according to the magnitude and nature of the particular positional shifting that is required to be 
corrected. If the positional shifting is limited to translation in x and y directions, the positional shifting Ax and Ay in x 
and y directions can be found by conducting a so-called template matching, using a representative region of a reference 
image, with respect to a neighborhood of the corresponding region of the test image. When the image involves exten- 
sion/shrinkage and rotation as its positional shitting, the positional correction can be carried out by using a so<;alled 

25 geometric transformation process. 

Figure 5 is a conceptual diagram of a typical geometric transformation process used for positional correction ac- 
cording to the conventional method, and Figure 6 is a flow chart of this process. An evaluation based on self differences 
is made with respect to a picture pattern of a reference image stored in reference image memory 102 obtained from 
an external appearance of a good product, and a desired number of regions (which are called as "cells" hereinafter) 

40 are selected as templates for computing positional shifting. The cells are selected from regions having large self dif- 
ferences such as regions including outlines. 

In the example given in Figure 5, six cells are selected. The image of the external appearance of the tested product 
is captured and stored in test image memory 1 03 as a test image. Once a full frame of the test image is entered in the 
test image memory 103. a template matching is carried out in a matching computation unit 104 with respect to each 

45 of the regions of the test image corresponding to the selected cells in the reference image, and the positional shifting 
(Ax^, Ay^) (e = 1, 2. 3, 4. 5, 6) at each of the cell positions (cell regions) is computed. According to these six positional 
shift values, the coefficients of the equation of geometric transformation (affine transformation in this case) or the 
transformation parameters are determined in a transformation parameter generating unit 105. 
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Xoj=:T11x^i+T12y^j+T13 (1) 
yoi = T21x,j+T22y,j+T23 (2) 



where 
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Xoj: x-address of a pixel of the reference image 
Voj: y-address of a pixel of the reference innage 

x^j: x-address of a pixel of the test image corresponding to that of the reference image with respect to the picture 
pattern thereon 

5 y-address of the pixel of the test image corresponding to that of the reference image with respect to the picture 

pattern thereon 
Tjj! transformation parameters 

The generated equation of transformation is stored in an address transformation unit 106. and the pixel addresses 
(^li- Yii) ^®st image which correspond to the predetermined pixel addresses (XQj. yoi) of the reference image SriS 

have been subjected to positional correction are generated by this equation of transformation. Thus, during the test 
procedure, the pixels at the addresses (x^j, y^j) of the test image that are to be compared with those pixels at the 
corresponding addresses (Xoj, yoi) of the reference image are supplied to a comparator 1 07, and the results of comparing 
the gradation values of the pixels on which positional shifting is corrected are produced so that the test image may be 
'5 evaluated as either good or bad. 

However, according to the conventional method, because the positional shifting is corrected with a geometrical 
process after the entire test image has been captured, and is based on the assumption that the extension/shrinkage 
and the translation of the picture pattern are uniform over the entire image, thus allowing a linear approximation to be 
applied to the two images to be compared, if there is any non-linear positional shifting or positional shifting involving 
localized extension/shrinkage, and the magnitude of extension/shrinkage varies from one place to another on the same 
imaqe the positional shifting cannot be totally eliminated even after a positional correction has been made, and the 
lost of the external appearance based on such image processing may erroneously judge the product to be defective 
even when the product is quite normal. Also, because positional shifting is computed and positional correction is made 
aticf the entire image has been captured, memory for storing the entire image is required, and it means a high cost 
lor the system, and a long processing time. 

BRIEF SUMMARY OF THE INVENTION 

In view of such problems of the prior art, a primary object of the present invention is to provide a method for image 
c^o mnichng which allows positional correction to be made even in the case of non-linear positional shifting which the 
pf lor Hft has been inadequate to deal with. 

A second object of the present invention is to provide a method for image matching which can carry out the positional 
correction substantially simultaneously as the capturing of the image on a real time basis without involving any exces- 
sive lime delay which would be otherwise required to capture the entire image before making any positional correction. 
35 A third object of the present invention is to provide a method for image matching which can eliminate the need for 

mernoiy for storing the entire frame of the picture image is eliminated, thereby reducing the processing time and the 
cost for hardware. 

A fourth object of the present invention is to provide a method for image matching which allows a stable positional 
matching to be achieved without any substantial error 
40 A fifth object of the present invention is to provide a method for image matching which can accommodate occasional 

errors in the input without producing any abnormal results in image matching. 

A sixth object of the present Invention is to provide a method for image matching which can reduce the so-called 
search area when conducting the template matching so that the processing time can be substantially reduced while 
the tolerance to the positional shifting Is increased. 
45 A seventh object of the present invention is to provide a method for image matching which can allows accumulation 

of positional shifts can be properly accounted for in the process of image matching. 

An eighth object of the present invention is to provide a system for testing products using this method. 
According to the present invention, these objections are accomplished by providing a method for positional match- 
ing of images, wherein image data of two images of an identical picture pattern is compared by using image data of 
so one of the images as reference Image data, and positional shifting between the two images are compensated for when 
determining an identity of the two images according computed differences between them, characterized by that: as a 
horizontal scan line or a group of horizontal scan lines are transferred in a sequential manner, a positional shifting 
computing process for the Images, and a positional matching compensation process based on the computed positional 
shifting are conducted for each of line blocks which are formed by a predetermined number of consecutive horizontal 
55 scan lines with respect to the two images of a nominally identical image pattern. 

According to the present Invention, a two dimensional image consisting of pixels arranged vertically as individual 
bits and horizontally along each scan line (gradation value data or binary data) is divided into individual line blocks 
consisting of a number of scan lines, and positional shifting is computed and positional correction is carried out for 
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each line block. Therefore, as opposed to the prior art, the need for mennory for storing the entire frame of the picture 
image is eliminated, and the processing time delay due to the need for storing the entire image can be substantially 
reduced while reducing the cost at the same time. Also, by carrying out the positional correction for each line block, it 
becomes possible to deal with non-linear positional shifts. 

5 A second aspect of the present invention provides a method for positional matching of images, wherein the posi- 

tional shifting computing process comprises the steps of: dividing each line block of the reference image into a number 
of cells; conducting a template matching on the reference image by using the cells as templates; selecting those celts 
which are effective in computing positional shifting by quantitatively evaluating the results of the template matching; 
classifying the selected cells into x-, y- and xy-cells as those which are effective in detecting the shifting in x, y and 

10 both X and y directions, respectively, and registering them as such; and conducting a template matching with res^e&t 
to the other image by using the registered selected cells as templates, and computing positional shifting of the other 
imago according to the result of the template matching. 

According to the method for positional matching of images provided by the second aspect of the present invention, 
when computing positional shifting for each line block by template matching, a novel algorithm is used wherein the 

^5 cells for template matching are selected by quantitatively evaluating the results of the plate matching conducted on 
the reference image so that the cells are automatically selected from areas which provide optimum template matching 
or the cells are automatically selected so as to be optimum templates. IV/lore specifically, template matching is conducted 
with respect to the reference image to select certain cells, and the cells thus selected according to the quantitative 
evaluation of the results of self template matching are classified into three kinds of cells, or x cells, y cells and xy cells 

20 according to the stability of template matching in the x and y directions, and are so registered. As a result, the cells 
are selected from optimum locations according each particular picture pattern (the distinction between good match 
and poor match is very clear, and pseudo matching does not occur within the search area), and a stable positional 
matching can be achieved without any substantial error 

A third aspect of the present invention provides a method for positional matching of images, wherein the positional 

25 shifting computing process comprises the steps of: conducing a template matching with respect to the other Image by 
using selected cells, obtained by dividing each of the line blocks of the reference image into a number of cells, as 
templates; quantitatively evaluating the results of the template matching as being either good or bad; and computing 
the positional shifting of the images according to the results of the template match ing which are evaluated as being good. 
According to the method for positional matching of images provided by the third aspect of the present invention, 

30 a novel algorithm is used wherein the results of template matching between the reference image and the test image 
are quantitatively evaluated, and good matching results are used for obtaining positional shifting while abnormal po- 
sitional shift values are rejected so that data which is likely to contain large errors is removed from the computation, 
and the positional matching can be accomplished in a short time in a stable fashion without involving any substantial 
errors. 

35 A fourth aspect of the present invention provides a method for positional matching of images, wherein if there are 

a plurality of cells serving as templates or selected cells in a line block of the reference image, there are a plurality of 
good matching results, and a plurality of incidences of positional shifting are obtained for a corresponding line block 
of the other image, then, the result of the positional shifting computing process Is given as an average of the incidences 
of positional shifting. 

40 According to the method for positional matching of images provided by the fourth aspect of the present invention, 

positional shift correction value for each line block Is given by averaging the positional shift values of the cells or the 
selected cells in the line block, positional shift correction process can be carried our for each line block in a stable 
manner, with a minimum error, by using a minimum amount of hardware, and In a short time. 

A fifth aspect of the present invention provides a method for positional matching of images, wherein If the positional 
45 shifting computed for a certain line block is greater than that of the preceding line block by more than a threshold value, 
the positional shifting of the certain line block Is set equal to a predetermined value. 

According to the method for positional matching of images provided by the fifth aspect of the present invention, it 
is possible to avoid any undesirable results which could occur If too much error is caused by pseudo matching, or to 
avoid the cases of excessive positional shift correction which are excessive In the sense that the amount of positional 
so shift correction far exceeds what are actually possible. ^ 

A sixth aspect of the present Invention provides a method for positional matching of images wherein when a tem- 
plate matching is conducted for a certain line block with respect to the other image by using the cells or the selected 
cells of the reference Image as templates, the address of each of the cells of the other Image is compensated for 
according to the positional shifting of the preceding line block with respect to the cell or the selected cell of the reference 
55 image at the corresponding address. 

According to the method for positional matching of images provided by the sixth aspect of the present Invention, 
when a template matching using the cells or the selected cells of the reference image is executed with respect to the 
corresponding area of the test Image, by correcting the addresses of the test image according to the positional shifting 
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in the preceding line block, it is possible to reduce the so-called search area when conducting the tennplate matching, 
and the processing time can be substantially reduced while the tolerance to the positional shifting can be increased. 

A seventh aspect of the present invention provides a method for positional matching of images, wherein the po- 
sitional matching compensation process for each of the line blocks is conducted according to the positional shifting of 
one of the line blocks preceding the current tine block or the positional shifting of one of the line blocks following the 
current line block. 

According to the method for positional matching of images provided by the seventh aspect of the present invention, 
accumulalion of positional shifting can be properly accounted for. 

An eighth aspect of the present invention provides a system for testing products by using the method for positional 
matching of images, comprising: image capturing means for sequentially entering a horizontal scan line or a grou^f 
horizontal scan lines of an image of a product to be tested; positional shifting computing means for connputing positional 
shifting of an image based on cell matching conducted for each lino block consisting of a prescribed number of adjacent 
horizontal scan lines of the test imago and a reference image; compensating means for compensating the positional 
shifting of the test image with respect to the reference image according to the positional shifting computed by the 
positional shifting computing means; difference computing means for comparing the test image, for which the positional 
shifting was compensated, with the reference image, and obtaining differences between them; and judging means for 
judging each product whether it is defective or not by determining the identity of the two images according to the 
computed differences between them; the positional shifting of each line block being compensated for before determining 
the identity of the two images. 

According to the system for testing products by using the method for positional matching of images provided by 
the eighth aspect of the present invention, the shape of the product and the picture pattern on the product can be 
evaluated on a real time basis, with a minimum amount of hardware, in a short lime, while tolerating the presence of 
non-linear image positional shifting. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Now the present invention is described in the following with reference to the appended drawings, in which: 

Figure 1 is a conceptual diagram of the first embodiment of the present invention to illustrate the principle of image 
position matching; 

Figure 2 is a block diagram for illustrating the structure and the mode of operation of the second embodiment of 
the present invention; 

Figure 3 is a flow chart showing the processing procedure of the above described embodiment; 

Figure 4 is a flow chart showing the processing procedure of the above described embodiment; 

Figure 5 is a flow chart showing the processing procedure of the above described embodiment of the product 

testing system and 

Figure 6 is a flow chart showing the processing procedure of the prior art. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Figure 1 is functional block diagram of a first embodiment of the present invention for describing the basic concept 
of the present invention A reference image 1 is stored in data storage means such as reference image memory, and 
is expressed by a two-dimensional matrix. The x-addresses of the two-dimensional matrix corresponds to the bit loca- 
tions along the scan line in image input scanning, and the y-addresses correspond to the line locations in image input 
scanning Every q lines of the image plane are bundled into a line block so that the entire image plane may be divided 

inton line blocks (LBOq LBO^. ... LBO^). Each line block isdivided into cells (Cqk. Ci^, .... C|k= ... C^^) each having 

a width of q bits. 

Once a reference image 1 is captured and stored, a matching arithmetic operation involving a local shifting is 
carried out in each of the cells on the reference image 1 , and those of the cells suitable for use as templates in an 
evaluation algorithm (which is described hereinafter) are selected and stored as selected cells. Normally, cells including 
clear picture patterns are selected as selected cells. In the case of Figure 1 , the shaded cells covering a cross-shaped 
picture pattern are selected as selected cells. The selected cells include x cells, y cells, and xy cells, x cells are those 
selected sells which are effective only for computing shifts in the x-direction. and include patterns involving relatively 
small changes in the y-direction (those marked by the left end down hatching pattern in the drawing). Likewise, y cells 
are those selected sells which are effective only for computing shifts in the y-direction, and include patterns involving 
relatively small changes in the x-direction (those marked by the right end down hatching pattern in the drawing), xy 
cells are those selected cells which are effective for computing shifts in both the x- and y<lirections, and involve image 
patterns involving large changes in both the x- and y-directions (those marked by the crossed hatching pattern in the 
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drawing). 

The addresses ot the selected cells are recorded, and a selected cell address generator 3 generates the addresses 
of the selected cells tor each line block In synchronisnn with the input ot a test image 2. For instance, when line block 
LB1 of the test image 2 is being entered, the selected cell address generator 3 generates the addresses (Xj, y|^) ot 

5 the selected cell C|,^ in the line block LBO,^ so that the pixel gradation value data of the selected cell Cj^ of the reference 
image 1 is forwarded to a cell matching arithmetic unit 4. 

Meanwhile the addresses (Xj, y^^) produced from the selected cell address generator 3 are forwarded to a cell 
address converter 5 to be each converted into an address (Xj +AXk.i, y^ +AY|^.i) having shifts of the preceding line 
block (AX|(..| , AYj(_i ) added thereto, and the converted addresses are produced as addresses of the test image 2 against 

^o which the cell matching is to be conducted. When k-0, the values of AX^^.-, and AYj^.i may be given as (AX,^.i=0%i1cJ 
aY,^.-,-0), as the shifts (AX|^.^- AX^ and AY,^.i= AY„) at the last line block of the previous test image {AX„, AYn)-^ or 
given as (AX|^.^=Aa and AYj^.^^Afi) according to the shifts (Aa, Ap) measured by different means. The pixel gradation 
value data of the test image 2 according to this address allocation arrangement is forwarded to the cell matching 
arithmetic unit 4, and a template matching using the selected cells Cj|^ for templates is conducted with respect to the 

?5 corresponding areas of the test image 2. At this point, because the addresses of the test image 2 are compensated 
for by the shifts of the preceding line blocks, the so-called search area for the template matching can be narrowed, 
and the necessary processing time can be significantly reduced. Also, the tolerance on the magnitude of shifts can be 
increased. 

The result of the template matching at the cell matching arithmetic unit 4 consists of a two<iimensional spatial 
20 distribution data (a two-dimensional map of cumulative values) formed by a two-dimensional matrix of cumulative 
values of pixel gradation differences of the ceil regions over the entire search area or a part thereof, a minimum value 
Sj of cumulative values of pixel gradation differences of the celt region, and the relative address (AXj|(, Ayji)" which gives 
rise to the minimum value Sj. The relative address (AXjj^, Ayj,) is an address relative to the address (x^, y,^) of the selected 
cell C|[^. The minimum value is compared with a threshold value in an evaluation unit 6. For instance, if S| > TH^ 
25 (where TH^ is a threshold value which can be determined either arbitrarily or by a computing process of image gradation 
difference cumulative values), the result of the template matching is determined to be reject. In the evaluation unit 6, 
a maximum value, a minimum value, a number of local maxima, a number of local minima, and an average value are 
computed from the two dimensional distribution data of the pixel gradation difference cumulative values, and are eval- 
uated so that the result of the template matching may be determined either good or bad. If the result is good, the 
30 address (Axj^, Ay||^) giving the minimum value Sj is forwarded to an averaging unit 7, and is averaged along with the 
other addresses (Ax^k, Ay^(^) giving minimum values Sr obtained by the template matching of other selected cells in the 
same line block. Thus, the positional shifting (Ax,^, Ay^^) for the line block LBI^ is given by the following equations. 

33 [Equation 1] 

^'^k- 1+1+ 

^ 1+1+ 

According to an example of an expanded averaging process, a comparison is made between the current positional 
shift (Ax,^, Ay,^) obtained by the averaging process expressed by the above equations with the positional shift (Ax,^.^. 

•45 Ay^.^) of the preceding line block. For instance, if Ax^ - Ax^.^ >1 , Isxy^ = ^^k'^ + Similarly, if AX|^ - Ax^.^ < -1 , Ax^ = Axj^.^ 
- 1 . By doing so, it is possible to avoid the inconveniences which would be otherwise caused when the positional shifts 
obtained by the nomnat averaging process contain large errors due to pseudo matching can be avoided. In other words, 
it is possible to avoid any excessive positional corrections resulting from detection of excessive positional shifts which 
are substantially greater than those possible under normal circumstances (whereas the positional shifts between ad- 

60 jacent line blocks are normally in the order of ±1 pixel). Also, the positional corr-oction can be carried out in a gradual 
manner so that the continuity of an image pattern from one line block to the next can be nnaintained in a smooth condition. 

When the selected cell consists of an x cell, Ayj^ is rejected. When the selected cell consists of a y cell, AXj|( is 
rejected. In either case, the selected cell is excluded from the averaging process. When the selected cells are absent 
or no positional shifting is detected from a given line block even when there are selected cells, the positional shift (Ax^, 

55 Ay^) of the current line block is set equal to the positional shift (AXi^.^, Ayj^.^) of the preceding line block. 

According to the positional shift (AXj^. Ay^) thus determined and stored in the memory 8, there Is generated the 
pixel address for comparison which has been subjected to positional shifting required for comparing the pixels. In other 
words, when executing the comparison with respect to the line block LB1|^ of the test image 2, the gradation value data 



6 



snnrin- <fp 



EP 0 755 024 A2 



10 



of the incident pixel of the test innage 2 is forwarded to the comparator 11 according to the address (x, y) of the test 
image 2 generated from the comparison pixel address generator 9. Meanwhile, the pixel address (x^, y^^) of the reference 
image 1 corresponding to the address of this incident pixel ot the test image is computed. 

When a positional shifting is detected in the picture pattern as a result of a cell matching, and the picture pattern 
extends to adjacent line blocks, the positional correction may be carried out with respect to the adjacent line blocks. 
For instance, when an address of the k-th line block is to be corrected, the amount of positional correction can be 
selected from those for any one the adjacent line blocks, or., k-1, k, k+1,... -th line blocks. 

Thus, the reference pixel address (x^, Vu) can be given by the following equations (assuming that the (k+1)-th 
address was selected). 

X, = X - AX,^, (3) 

15 = y - AY,,, (4) 

where, if k = n (or in case of the last line block), 
20 = X - AX„ 

If required, the equations (3) and (4) can be also given by the following equations (when the k-th line block is selected). 

x, = X - AX^ 
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According to the reference pixel address {x^, y^) obtained above, the gradation value data of the reference pixel 
of the reference image is forwarded to the comparator 11 . In this manner, the comparative computation is executed 
with respect to the pixels which are positionally matched between the reference image and the test image. 

Now a second embodiment ot the method of image matching according to the present'invention is described in 
the following with reference to the appended drawings. Figure 2 is a functional block diagram of the second embodiment. 

In the drawing, REM denotes reference image memory which is capable of retaining a frame of the reference 
image. LBM denotes line block buffer memory, and in the illustrated embodiment, this memory retains three line blocks. 
However, this number can be freely changed according to the requirements such as the search range and the process- 
ing speed in the template matching. The reference image memory REM, the line block buffer memory LBM, and so 
forth can consist of random access memory or first-in first-out (FIFO) memory as required. 

A cell matching arithmetic unit CMU receives selected cell images serving as templates from A input, and the 
image to be searched from B input, the image to be searched being supplied by a selector SEL which is described 
hereinafter The output of the cell matching arithmetic unit CMU consists of a group of pixel difference cumulative 
values over the entire search range for each individual shifting, and a positional shift value (relative address) which 
gives rise to the minimum value S of the cumulative values. SEL denotes the selector which supplies, to the B input 
of the cell matching arithmetic unit CMU, a reference image during the process of registering the reference image, and 
a test image during the process of testing. ADD denotes a cell address converter which generates the addresses of 
the test image on which the template matching is to be executed according to the addresses supplied by a selected 
cell register table SCT which is described hereinafter 

MEU denotes a matching error evaluation unit which determines the result of the template matching for each of 
the selected cells by evaluating the magnitude of the minimum value S which reflects the extent of mis-match during 
the testing process. The reference of evaluation may be based on a freely selected threshold value, or a threshold 
value which is determined statistically according to the group ot cumulative values. Alternatively it may also be deter- 
mined by a multi-purpose arithmetic unit MPU. 

AVU denotes an averaging computation unit which comprises a computation circuit for finding an average value 
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ot the positional shifts of a plurality of cells in each line block during the testing process, and, optionally, a connparator 
for comparing the positional shifts with those of the preceding line block. DMA denotes a positional shift nnemory which 
stores at least the positional shifts of the current line block and the preceding line block. 

CEU denotes a selected cell evaluation unit which determines the selected cells and classify them into the three 
categories according to the spatial distribution of the group of cumulative values and the magnitudes of the cumulative 
values other than the minimum value S. CST denotes a cell evaluation order determination table which determines 
which of the cells should be evaluated and the order of evaluating those cells that are going to be evaluated when 
evaluating all the cells of each line block for selective purpose by self template matching in a sequential manner 
Normally, to avoid any local unevenness in the selection of cells, the order is assigned in a random fashion. Because 
it is sometimes advantageous to control the locations of the selected cells according to the nature of the picture patl&ffi, 
the locations of the cells to be evaluated may be allowed to be freely determined by the multi-purpose arithmetic unit 
MPU which is described hereinafter. SCT denotes a selected cell register table corresponding to the selected cell 
address generator, and performs the function of registering the addresses and the classifications of all the selected 
cells following the input of the reference image and the subsequent process of cell selection. 

MAG denotes a comparative pixel address generator which generates and outputs addresses in synchronism with 
the start timing and the line start timing of the input image, and a pixel clock. SUB denotes a subtracter corresponding 
to the comparative pixel address converter, and generates addresses by subtracting the positional shifts stored in the 
positional shifting memory DAM from the comparative pixel addresses generated from the comparative pixel address 
generator which is described hereinafter, during the testing process, and uses them as the comparative pixel addresses 
of the reference image memory REM. PCU denotes a pixel comparator which executes a comparative arithmetic op- 
eration with respect to the pixel gradation value data of the pixels which are stored in the reference image memory 
REM or memory (not shown in the drawing) having a same address space as REM and have been subjected to posi- 
tional correction with the pixels stored in the line block buffer memory LBM. 

MPU denotes a multi-purpose arithmetic unit which sets the operating conditions and the data for the functional 
blocks, and carries out the control of the overall operation of the functional blocks. 

Now the mode of operation of the above embodiment is described in the following. Figure 3 is a flow chart for 
describing the mode of operation. 

In the initial step of registering the reference, the input image is stored and retained in the reference image memory 
REM. Then, the cells at the addresses designated by the cell evaluation order setup table CST are transferred from 
the reference image memory REM to the cell matching arithmetic unit CMU where the cell matching is carried out. 
Cumulative values of pixel differences are forwarded to the selected cell evaluation unit CEU to be evaluated and 
classified accordingly, and the selected cells are registered at the selected cell registration table SCT This completes 
the step of registering the reference. 

Thereafter, the test image is supplied to the line block buffer memory LBM one line block at a time, and the part 
of the lest image at the test image addresses generated by converting the addresses designated by the selected cell 
registration table SCT is forwarded to the cell matching arithmetic unit CMU via the selector SEL. Meanwhile, the 
selected cells at the addresses designated by the selected cell registering table SEL is forwarded to the cell matching 
arithmetic unit CMU from the reference image memory REM. The template matching is then carried out with respect 
to the selected cells, and data such as cumulative pixel difference values is forwarded to the matching error evaluation 
unit MEU to bo evaluated thereby. The results of evaluation with respect to a plurality of selected cells are averaged 
by the averaging arithmetic unit AVU. and the positional shifting in the corresponding line block is computed, and is 
forwarded to the positional shifting memory DAM to be retained therein. 

The comparison of pixels in the reference and test images is executed as described in the following. A time delay 
is given to the line block of the test image which has been stored in the line block buffer memory LBM until the necessary 
positional shifts are obtained, and is forwarded to the pixel comparator PCU in a sequential manner The pixels of the 
reference image stored in the reference image memory REM corresponding to the above line block are supplied to 
the pixel comparator PCU. and are compared with the corresponding pixels of the test image in the pixel comparator 
PCU. The addresses of the pixels for comparison are generated by the comparison pixel address generator MAG, and 
are forwarded to the line block buffer memory LBM and the reference image memory REM. The addresses forwarded 
to the reference image memory REM are subjected to a positional shifting corroction computing process in the sub- 
tractor SUB. The image to be compared with the test image is not limited to the image stored in the reference image 
memory REM, but may also be an image stored externally. Therefore, depending on the case, the output from the 
subtractor SUB may be supplied to outside the system as positional correction values. 

In this embodiment, the pixel addresses of the reference image memory REM were corrected by using the pixel 
addresses of the line block buffer memory LBM as a reference, but because the pixel addresses are treated as relative 
addresses between the two images, the reference of the addresses may be set on the pixel addresses of the reference 
image memory REM. 

Now an embodiment of the product testing device using the image positional matching method of the present 
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invention is described in the following. 

Figure 4 is a drawing for illustrating both the structure and mode of operation of this ennbodinnent. Nunneral 21 
denotes a camera for capturing the image of the external appearance of the product 22 to be tested, and the image 
may comprise whatever is printed on this product. The image captured by the camera 21 is sequentially forwarded to 

5 an Image positional matching unit 23 by each line block. The structure of the positional matching unit 23 may be as 
illustrated in Figures 1 and 2, and the positional matching unit 23 computes the differences between the reference 
image which Is registered in advance and the input image from the camera 21 while making corrections based on the 
positional matching for each line block, and generates a minimum value as a difference value. Numeral 24 denotes a 
determination unit for determining if the difference value produced from the image positional matching unit is greater 

10 than the threshold value or not to determine if there are any defects in the external appearance of the product 2^^r 
the contents of the print thereon. 

According to the above described embodiments of the present invention, the invention was applied to the testing 
of products which is based on the comparison of the pixels of two images. But, the present invention can be equally 
applied to other applications where positional shifting is required to be measured quickly and at high precision such 

1^ as the measurement of fluctuations of speed, lateral deviation, and positional shifting in the field of conveying an object 
at a fixed speed. 

As can be appreciated from the above description, according to the algorithm of the image positional matching 
method of the present invention which sequentially tests the positional shifts for each of the line blocks dividing the 
image into a number of equal parts, because the positional matching between the reference image and the test image 

20 is executed for each line block in both X and Y directions even when the test image includes non-linear shifts and 
elongation/shrinkage, positional matching involving a minimum error can be accomplished, and a high level of reliability 
can be achieved. Also, because the positional shifts can be computed within a limited delay time corresponding to only 
a few line blocks (Because each line block normally consists of a few to a few dozen lines, and the Image is typically 
hundreds or thousands lines wide, this time delay is estimated to be only a fraction of the time required for inputting 

25 the entire image.), the comparative computation of the pixels can be executed substantially simultaneously as the 
inputting of the image, and the time pehod required for the testing process can be made extremely short. Because the 
testing process could not be started until the entire image has been read according to the conventional method, the 
present invention can achieve a substantial reduction in the processing time. Furthermore, because the test image 
becomes unnecessary as soon as the computing of the positional shifting and the comparison of the pixeis are com- 

30 pleted, the test image can be renewed for each tine block, and only a few line blocks of the test image are required to 
be retained at a time. Thus, the image memory required for storing the test image can be reduced to a fraction of that 
required for storing the entire image. 

In the image positional matching method of the present Invention, it the results of the template matching with 
respect to the reference image are quantitatively evaluated in selecting the cells serving as the templates in the template 

35 matching for computing positional shifting, and the selected cells are registered after classifying them into three types, 
a sufficient number of selected cells including picture patterns suitable for computing positional shifting can be obtained, 
and positional matching can be achieved in a stable manner and with a minimum error. 

Also, in the image matching method of the present invention, if the positional shifting is carried out by selectively 
using only those positional shifting values by the evaluation of the results of positional matching with respect to the 

40 template matching in the test image for computing the positional shifting, abnormal positional shifting values are re- 
jected, and positional matching can be achieved in a stable manner and with a minimum error. 

In the image position matching method, when the positional shifting values obtained from a plurality of cells are 
averaged In the template matching of the test image to compute the positional shifting values, abnormal positional 
shifting values are rejected, and positional matching can be achieved in a stable manner and with a minimum error 

45 In the method for positional matching of images according to the present invention, if the positional shifting com- 

puted for a certain line block is greater than that of the preceding line block by more than a threshold value in the step 
of computing the positional shifting of the image, the positional shifts of the certain line block may be set equal to a 
predetermined value. In such a case, it is possible to avoid any undesirable results which could occur it too much error 
is caused by pseudo matching, or to avoid the cases of excessive positional shift correction which are excessive in 

50 the sense that the amount of positional shift correction far exceeds what are actually possible, and the positional shift 
correction can be effected in a gradual manner so that the connection of the image between line blocks may be kept 
smooth even after positional shift correction has been executed. 

In the method for positional matching of images according to the present invention, if a template matching is 
conducted for a certain fine block with respect to the test image by using the cells or the selected cells of the reference 

55 image as templates, the address of each of the cells of the test image may be compensated for according to the 
positional shifting of the preceding line block with respect to the cell or the selected cell of the reference image at the 
corresponding address, the so-called search area when conducting the template matching can be narrowed, and the 
processing time can be substantially reduced while the tolerance to the positional shifting can be increased. 
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In the method for positional matching of images according to the present invention, if the positional matching 
compensation process for each of the line blocks is conducted according to the positional shifting of one of the line 
blocks preceding the current line block or the positional shifting of one of the line blocks following the current line block, 
the accumulation of positional shifting can be property accounted for. 

5 As for the system for testing products according to the present invention, it employs image capturing means for 

sequentially entering a horizontal scan line or a group of horizontal scan lines of an image of a product to be tested, 
positional shifting computing means for computing positional shifting of an image based on cell matching conducted 
for each line block consisting of a prescribed number of adjacent horizontal scan lines of the lest image and a reference 
image, compensating means for compensating the positional shifting of the test image with respect to the reference 

^0 image according to the positional shifting computed by the positional shifting computing means, and difference cBrf?- 
puting means for comparing the test image, for which the positional shifting was compensated, with the reference 
image, and obtaining differences between them, so that each product may bo judged whether it is defective or not by 
determining the identity of the two images according to the computed differences between them. Therefore, the shape 
of the product and the picture pattern on the product can be evaluated on a real time basis, with a minimum amount 

^5 of hardware, in a short time, while tolerating the presence of non-linear image positional shifting. 

Claims 

20 1. A method for positional matching of images, wherein image data of two images of an identical picture pattern is 
compared by using image data of one of the images as reference image data, and positional shifting between the 
two images are compensated for when determining an identity of the two images according computed differences 
between them, characterized by that: 

as a horizontal scan line or a group of horizontal scan lines are transferred in a sequential manner, a positional 

25 shifting computing process for the images, and a positional matching compensation process based on the com- 

puted positional shifting are conducted for each of line blocks which are formed by a predetermined number of 
consecutive horizontal scan lines with respect to the two images of a nominally identical image pattern. 

2. A method for positional matching of images according to claim 1 , wherein the positional shifting computing process 
30 comprises the steps of: 

dividing each line block of the reference image into a number of cells; 
conducting a template matching on the reference image by using the cells as templates; 
selecting those cells which are effective in computing positional shifting by quantitatively evaluating the results 
•55 of the template matching; 

classifying the selected cells into x-, y- and xy-cells as those which are effective in detecting the shifting in x, 
y and both x and y directions, respectively, and registering them as such; and 

conducting a template matching with respect to the other image by using the registered selected cells as 
templates, and computing positional shifting of the other image according to the result of the template matching. 

40 

3. A method for positional matching of images according to claim 1 . wherein the positional shifting computing process 
comprises the steps of: 

conducing a template matching with respect to the other image by using selected cells, obtained by dividing 
^5 each of the line blocks of the reference image into a number of cells, as templates; 

quantitatively evaluating the results of the template matching as being either good or bad; and 

computing the positional shifting of the images according to the results of the template matching which are 

evaluated as being good. 

50 4. A method for positional matching of images according to claim 3, wherein if there are a plurality of cells serving 
as templates or selected cells in a line block of the reference image, there are a plurality of good matching results, 
and a plurality of incidences of positional shifting are obtained for a corresponding line block of the other image, 
then, the result of the positional shifting computing process is given as an average of the incidences of positional 
shifting. 



55 



A method for positional matching of images according to claim 1, wherein if the positional shifting computed for a 
certain line block is greater than that of the preceding line block by more than a threshold value, the positional 
shifting of the certain line block is set equal to a predetermined value. 
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A method for positional matching of images according to claim 2. wherein when a template matching is conducted 
for a certain line block with respect to the other image by using the celts or the selected cells of the reference 
image as templates, the address of each of the cells of the other image is compensated for according to the 
positional shifting of the preceding line block with respect to the cell or the selected cell of the reference image at 
the corresponding address. 

A method for positional matching of images according claim 1, wherein the positional matching compensation 
process for each of the line blocks is conducled according to the positional shifting of one of the line blocks pre- 
ceding the current tine block or the positional shifting of one of the line blocks following the current line block. 

A system for testing products by using the method for positional matching of imagesrcomprising: 

image capturing means for sequentially entering a horizontal scan line or a group of horizontal scan lines of 
an image of a product to be tested; 

positional shifting computing means for computing positional shifting of an image based on cell matching 
conducted for each line block consisting of a prescribed number of adjacent horizontal scan lines of the test 
image and a reference image; 

compensating means for compensating the positional shifting of the test image with respect to the reference 
image according to the positional shifting computed by the positional shifting computing means; 
difference computing means for comparing the lest image, for which the positional shifting was compensated, 
with the reference image, and obtaining differences between them; and 

judging means for judging each product whether it is defective or not by determining the identity of the two 
images according to the computed differences between them; 

the positional shifting of each line block being compensated for before determining the identity of the two 
images. 
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FIG. 4 
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(57) The present invention provides an image 
matching method which allows non-linear image posi- 
tional shifts to be corrected on a real time basis, and an 
economical product testing system using this method. 
A two dimensional test image, consisting of pixels ar- 
ranged as a matrix of bits distributed in both the vertical 
direction and the horizontal scanning direction, and an 
associated reference image are subjected to template 



matching at a ceil matching arithmetic unit for each of 
line blocks consisting of a plurality of scan lines. The 
result of the template matching are quantitatively eval- 
uated at an evaluation unit as being good or bad, and 
positional shifts are sequentially computed according to 
good evaluation results. Based on the positional shifts 
thus obtained, the comparative pixel addresses are ac- 
cordingly corrected before the two images are com- 
pared at the comparator 
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